Skip to content

Remove properties and cross_validation FilePaths from metadata#2869

Merged
allisterakun merged 18 commits intodevfrom
remove_properties_fp_from_metadata
Mar 25, 2026
Merged

Remove properties and cross_validation FilePaths from metadata#2869
allisterakun merged 18 commits intodevfrom
remove_properties_fp_from_metadata

Conversation

@allisterakun
Copy link
Collaborator

@allisterakun allisterakun commented Mar 19, 2026

Removes properties and cross-validation file path definitions from metadata JSON files. Properties file paths are now defined as a module-level constant in input_manager.py, and cross-validation file paths are moved to the task configuration JSON files.

Context

Issue(s) closed by this pull request: closes #2859.

What

  • Removed the "properties" file blob entry from all metadata JSON files (freestall_e2e_metadata.json, no_animal_e2e_metadata.json, open_lot_e2e_metadata.json, example_freestall_dairy_metadata.json, example_no_animal_metadata.json, example_open_lot_metadata.json, available_simulations_metadata.json, end_to_end_testing_tm_metadata.json, herd_init_metadata.json, update_end_to_end_testing_tm_metadata.json, task_manager_metadata.json).
  • Removed "properties" from REQUIRED_FILE_BLOBS in input_manager.py.
  • Added a module-level constant PRROPERTIES_FILE_PATHS in input_manager.py that maps property file names to their Path objects (default.json, tasks_properties.json, commodity_properties.json).
  • Updated _load_properties in InputManager to iterate over the hardcoded PRROPERTIES_FILE_PATHS instead of reading paths from the metadata.
  • Removed "cross-validation" key from all metadata JSON files.
  • Added "cross_validation_file_paths" as a new field in task configuration JSON files (available_simulation_tasks.json, end_to_end_testing_task.json, example_freestall_task.json, example_no_animal_task.json, example_open_lot_task.json, single_run.json).
  • Updated start_data_processing to accept a cross_validation_file_paths: list[str] | None parameter and pass it to _cross_validate_data.
  • Updated _cross_validate_data to receive cross_validation_file_paths as a parameter instead of reading it from self.__metadata.
  • Updated TaskManager.start and handle_input_data_audit to pass cross_validation_file_paths through to start_data_processing.
  • Added "default" to optional_array_property_keys in DataValidator._metadata_array_validator.
  • Updated input/metadata/properties/tasks_properties.json with the cross_validation_file_paths property, where it is a nullable array of strings with default being null.
  • Updated unit tests in test_input_manager.py and test_task_manager.py.

Why

Previously, properties and cross-validation file paths were embedded inside each metadata JSON file. This led to issue #2859. Properties file paths are the same across all simulations, so hardcoding them as a constant simplifies metadata files and eliminates redundancy. Moving cross-validation file paths to the task configuration files gives each task explicit control over which cross-validation rules apply, which is a more appropriate location since different tasks may require different validation rules.

How

For properties, the _load_properties method now reads from the PRROPERTIES_FILE_PATHS constant dictionary instead of extracting paths from self.__metadata["files"]["properties"]. The "properties" entry is no longer needed in metadata files and has been removed, along with the del self.__metadata["files"]["properties"] cleanup step.

For cross-validation, the "cross-validation" key is removed from metadata files entirely. Instead, each task JSON file now includes a "cross_validation_file_paths" field (a list of file path strings, or an empty list). This value flows through TaskManager.handle_input_data_auditargs["cross_validation_file_paths"]InputManager.start_data_processing_cross_validate_data_load_cross_validation.

Test plan

  • Updated unit tests in test_input_manager.py to reflect the new cross_validation_file_paths parameter in start_data_processing and the removal of properties from metadata.
  • Updated unit tests in test_task_manager.py to pass cross_validation_file_paths in the appropriate call assertions.

Input Changes

  • Updated: input/metadata/properties/tasks_properties.json.
  • Added: "cross_validation_file_paths" field to all task JSON files (available_simulation_tasks.json, end_to_end_testing_task.json, example_freestall_task.json, example_no_animal_task.json, example_open_lot_task.json, single_run.json).
  • Deleted: "properties" file blob entry from all metadata JSON files.
  • Deleted: "cross-validation" key from all metadata JSON files.

Output Changes

  • N/A

Filter

@github-actions
Copy link
Contributor

Current Coverage: %

Mypy errors on remove_properties_fp_from_metadata branch: 1213
Mypy errors on dev branch: 1213
No difference in error counts

@github-actions
Copy link
Contributor

🚨 Please update the changelog. This PR cannot be merged until changelog.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Some tests have failed.

@github-actions
Copy link
Contributor

Current Coverage: 99%

Mypy errors on remove_properties_fp_from_metadata branch: 1213
Mypy errors on dev branch: 1213
No difference in error counts

@github-actions
Copy link
Contributor

🚨 Please update the changelog. This PR cannot be merged until changelog.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

@morrowcj morrowcj self-requested a review March 23, 2026 15:52
Copy link
Collaborator

@ew3361zh ew3361zh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice work Allister.

"feed_storage_configurations",
"feed_storage_instances",
}
PRROPERTIES_FILE_PATHS: dict[str, Path] = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think docstrings would be helpful here? I know we're not doing it anywhere else in this file but we do it in manure files for file-level constants. Just a thought.

allisterakun and others added 3 commits March 24, 2026 10:28
@github-actions
Copy link
Contributor

Current Coverage: 99%

Mypy errors on remove_properties_fp_from_metadata branch: 1214
Mypy errors on dev branch: 1214
No difference in error counts

@github-actions
Copy link
Contributor

🚨 Please update the changelog. This PR cannot be merged until changelog.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

@allisterakun allisterakun marked this pull request as ready for review March 24, 2026 16:44
@github-actions
Copy link
Contributor

Current Coverage: 99%

Mypy errors on remove_properties_fp_from_metadata branch: 1214
Mypy errors on dev branch: 1214
No difference in error counts

@github-actions
Copy link
Contributor

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

@github-actions
Copy link
Contributor

Current Coverage: 99%

Mypy errors on remove_properties_fp_from_metadata branch: 1215
Mypy errors on dev branch: 1215
No difference in error counts

@github-actions
Copy link
Contributor

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

@allisterakun allisterakun merged commit 4378126 into dev Mar 25, 2026
1 check passed
@allisterakun allisterakun deleted the remove_properties_fp_from_metadata branch March 25, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Running external inputs fails, not respecting external properties paths.

3 participants